feat: heuristic-based model routing for cost optimization#11272
feat: heuristic-based model routing for cost optimization#11272roomote[bot] wants to merge 2 commits intomainfrom
Conversation
Adds an experimental feature that dynamically routes API calls to a lighter/cheaper model when the task is in an information-gathering phase. Addresses Issue #11269 - Choose Model Dynamically Based on Request. ## How it works - New experiment flag: modelRouting (disabled by default) - New setting: modelRoutingLightModelId - the cheaper model ID to use - ModelRouter tracks tool usage per API turn - If previous turn only used "read" group tools (read_file, list_files, search_files, codebase_search), the next API call uses the light model - Edit, command, browser, and MCP tools always use the primary model - First turn always uses the primary model ## Changes - packages/types/src/experiment.ts: Add modelRouting experiment ID - packages/types/src/global-settings.ts: Add modelRoutingLightModelId - src/shared/experiments.ts: Add MODEL_ROUTING config - src/core/task/ModelRouter.ts: New heuristic-based model router - src/core/task/Task.ts: Integrate ModelRouter into task lifecycle - src/core/task/__tests__/ModelRouter.spec.ts: 35 tests (all passing)
The latest commit fixes the empty-response handler restoration and removes the
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| // Model routing: temporarily swap to light model if heuristics say so | ||
| let primaryApiHandler: typeof this.api | undefined | ||
| { | ||
| const routingState = await this.providerRef.deref()?.getState() | ||
| if ( | ||
| ModelRouter.isEnabled(routingState?.experiments, routingState?.modelRoutingLightModelId) && | ||
| this.modelRouter.shouldUseLightModel() | ||
| ) { | ||
| const lightHandler = ModelRouter.buildLightModelHandler( | ||
| this.apiConfiguration, | ||
| routingState!.modelRoutingLightModelId!, | ||
| ) | ||
| if (lightHandler) { | ||
| primaryApiHandler = this.api | ||
| this.api = lightHandler | ||
| console.log( | ||
| `[Task#${this.taskId}] Model routing: using light model "${routingState!.modelRoutingLightModelId}" for this turn`, | ||
| ) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Bug: primaryApiHandler is not restored to this.api on two code paths, permanently losing the primary handler for the rest of the task:
- Stream failure (catch block, ~line 3280): When the light model's stream throws (rate limit, network error, etc.), the catch block calls
continueto retry without restoringthis.api. On the next iteration,primaryApiHandler = this.apicaptures the light handler as "primary", so even a successful retry restores to the wrong handler. - Empty API response (~line 3673): The restoration block at line 3626 is inside
if (hasTextContent || hasToolUses). When the API returns no content, the else branch retries without restoring, causing the same permanent loss.
The fix is to move the handler restoration (and endTurn()) into a finally-like position that executes regardless of path -- either a try/finally wrapping the stream + post-processing, or explicit restoration in both the catch block (before continue) and the empty-response else branch.
Fix it with Roo Code or mention @roomote and request a fix.
src/core/task/ModelRouter.ts
Outdated
| if (!experimentsConfig || !lightModelId || lightModelId.trim() === "") { | ||
| return false | ||
| } | ||
| return experiments.isEnabled(experimentsConfig, EXPERIMENT_IDS.MODEL_ROUTING as any) |
There was a problem hiding this comment.
Nit: EXPERIMENT_IDS.MODEL_ROUTING is already typed as "modelRouting" which satisfies ExperimentId, so the as any cast is unnecessary. It also silences the compiler if MODEL_ROUTING were ever removed or renamed in the experiment IDs, which is exactly the kind of error you'd want caught at build time.
| return experiments.isEnabled(experimentsConfig, EXPERIMENT_IDS.MODEL_ROUTING as any) | |
| return experiments.isEnabled(experimentsConfig, EXPERIMENT_IDS.MODEL_ROUTING) |
Fix it with Roo Code or mention @roomote and request a fix.
|
@roomote can you pleaE explain where and how I would set the light model id on the ui based on the code changes here. Thanks |
Hey @farazoman -- looked through the code changes and here's the situation: What the PR currently provides in the UIThe experiment toggle will automatically show up in Settings > Experimental tab as a checkbox labeled "Model Routing" (or the raw translation key, since no locale strings were added -- see below). This is because What's missing from the UIThere is no input field for
But no text input component was added to the webview UI to actually let users enter a light model ID. So currently, there's no way to configure this value through the UI. Additionally, no translation keys for What would need to be addedTo make this fully functional in the UI, you'd need:
Hope that clarifies things! |
@roomote can you add this code you suggested to the active PR. Thanks |
Added the model routing UI (translation keys, |
- Add translation keys for MODEL_ROUTING in locales/en/settings.json - Create ModelRoutingSettings component with toggle and light model ID input - Wire up MODEL_ROUTING special-case in ExperimentalSettings.tsx - Add modelRoutingLightModelId state binding in SettingsView.tsx - Fix bug: restore this.api to primary handler on all paths (stream failure, empty-response retry, catch) not just the happy path - Fix nit: remove unnecessary as any cast in ModelRouter.isEnabled()
2 participants
This PR attempts to address Issue #11269 - Choose Model Dynamically Based on Request.
Summary
Adds an experimental, rule/heuristic-based model routing system that dynamically routes API calls to a lighter (cheaper) model when the task is in an information-gathering phase. This is the first iteration, focusing on simplicity as discussed in the issue comments.
How it works
modelRouting(disabled by default, opt-in via Experimental Settings)modelRoutingLightModelId- the model ID for the cheaper model (must be from the same provider)ModelRoutertracks tool usage per API turn:read_file,list_files,search_files,codebase_search), the next API call uses the light modelask_followup_question,update_todo_list, etc.) are tier-neutral and don't affect classificationChanges
packages/types/src/experiment.tsmodelRoutingexperiment IDpackages/types/src/global-settings.tsmodelRoutingLightModelIdsettingpackages/types/src/vscode-extension-host.tssrc/shared/experiments.tsMODEL_ROUTINGexperiment configsrc/core/task/ModelRouter.tssrc/core/task/Task.tssrc/core/webview/ClineProvider.tssrc/shared/__tests__/experiments.spec.tssrc/core/task/__tests__/ModelRouter.spec.tsDesign decisions aligned with issue discussion
Future iterations
Feedback and guidance are welcome!
Important
Introduces heuristic-based model routing for cost optimization, adding new settings, UI components, and tests.
ModelRouterclass inModelRouter.tsfor heuristic-based model routing.modelRoutingexperiment flag andmodelRoutingLightModelIdsetting inexperiment.tsandglobal-settings.ts.ModelRouterintoTask.tsto dynamically switch models based on tool usage.ExperimentalSettings.tsxandModelRoutingSettings.tsx.SettingsView.tsxto handle new model routing settings.settings.json.ModelRouter.spec.tsfor testingModelRouterfunctionality.experiments.spec.tsto includemodelRoutingexperiment.This description was created by
for 37c8e2a. You can customize this summary. It will automatically update as commits are pushed.